Transmit rate pacing of large network traffic bursts to reduce jitter, buffer overrun, wasted bandwidth, and retransmissions

ABSTRACT

A system, method and medium is disclosed which includes selecting, at a software component of a network traffic management device, a first bucket having a first predetermined transmit time. The disclosure includes populating one or more selected data packet descriptors associated with one or more corresponding data packets in the first bucket. The disclosure includes releasing the first bucket to a hardware component of the network traffic management device, wherein the hardware component processes the one or more data packet descriptors of the first bucket for the first predetermined transmit time.

FIELD

This technology relates to a system and method for pacing networktraffic between network devices.

BACKGROUND

It is very common for a modern server to transmit large blocks of datain one burst to a single destination where the network path to thedestination has much lower bandwidth than the really large bandwidth ofthe server's path within the data center and the first few hops alongthat path. Often these bursts are in response to a request for data, andthe data is read from a storage medium (e.g., a disk) in a large blockto help amortize the cost of the read operation. The large chunk of datais then dumped by the server's OS into the network at full speed, addinglatency for other traffic following some portion of the same paththrough the network. This latency is unnecessary since other trafficcould easily have been interleaved with the packets of the burst if thepackets of the burst were spaced in time to match the true bandwidth ofthe full path to the destination. Further, such bursts can result inloss of some of the burst data due to overruns in of the buffering inthe network path to the destination. Such losses necessitateretransmission of some of the large chunk of data—effectively reducingthe gains that were hoped to be achieved by the batching of the readoperation into a large chunk and reducing the overall throughput of theserver. If the packets in these bursts were, instead, spread out in timeat a pace matching the full network path data rate, both of theseproblems are easily solved

What is needed is a system and method which overcomes thesedisadvantages.

SUMMARY

In an aspect, a method for pacing data packets from one or more sessionscomprises selecting, at a software component of a network trafficmanagement device, a first bucket having a first predetermined transmittime. The method comprises populating one or more selected data packetdescriptors associated with one or more corresponding data packets inthe first bucket. The method comprises releasing the first bucket to ahardware component of the network traffic management device, wherein thehardware component processes the one or more data packet descriptors ofthe first bucket for the first predetermined transmit time.

In an aspect, a processor readable medium having stored thereoninstructions for pacing data packets from one or more sessions,comprising machine executable code which when executed by at least oneprocessor and/or network interface a network traffic management deviceto perform a method comprising selecting a first bucket having a firstpredetermined transmit time; populating one or more selected data packetdescriptors associated with one or more corresponding data packets inthe first bucket; releasing the first bucket to a hardware component ofthe network traffic management device, wherein the hardware componentprocesses the one or more data packet descriptors of the first bucketfor the first predetermined transmit time.

In an aspect, a network traffic management device comprises a memorycontaining non-transitory machine readable medium comprising machineexecutable code having stored thereon instructions for pacing datapackets from one or more sessions. A network interface configured tocommunicate with one or more servers over a network. A processor coupledto the network interface and the memory, the processor configured toexecute the code which causes the processor to perform, with the networkinterface, a method comprising: selecting a first bucket having a firstpredetermined transmit time; populating one or more selected data packetdescriptors associated with one or more corresponding data packets inthe first bucket; releasing the first bucket to a hardware component ofthe network traffic management device, wherein the hardware componentprocesses the one or more data packet descriptors of the first bucketfor the first predetermined transmit time.

In one or more of the above aspects, the method performs comprisesselecting, at the software component, a second bucket having a secondpredetermined transmit time that is the same as the first transmit timeof the first bucket; populating one or more selected data packetdescriptors associated with one or more corresponding data packets inthe second bucket; releasing the second bucket to the hardware componentof the network traffic management device, wherein the hardware componentprocesses the one or more data packet descriptors of the second bucketfor the second predetermined transmit time quanta.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an example system environment that includes anetwork traffic management device in accordance with an aspect of thepresent disclosure.

FIG. 2A is a block diagram of the network traffic management device inaccordance with an aspect of the present disclosure.

FIG. 2B illustrates a block diagram of the network interface inaccordance with an aspect of the present disclosure.

FIG. 2C illustrates further details of the network traffic managementdevice in accordance with an aspect of the present disclosure.

FIG. 3 illustrates an example transmit ring or bucket in accordance withan aspect of the present disclosure.

FIG. 4 illustrates a time line of an example transmit ring or bucket inaccordance with an aspect of the present disclosure.

FIG. 5 illustrates a block diagram of hardware based time enforcement inperformed by a high speed bridge (HSB) priority mechanism in accordancewith an aspect of the present disclosure.

FIG. 6 illustrates an example transmit time line in accordance with anaspect of the present disclosure.

FIG. 7 illustrates a flow chart of the software implementation ofpopulating buckets with data packet descriptors in accordance with anaspect of the present disclosure.

FIG. 8 illustrates a flow chart of the software implementation ofpopulating buckets with data packet descriptors in accordance with anaspect of the present disclosure.

While these examples are susceptible of embodiment in many differentforms, there is shown in the drawings and will herein be described indetail preferred examples with the understanding that the presentdisclosure is to be considered as an exemplification and is not intendedto limit the broad aspect to the embodiments illustrated.

DETAILED DESCRIPTION

FIG. 1 is a diagram of an example system environment that includes anetwork traffic management device in accordance with an aspect of thepresent disclosure. The example system environment 100 includes one ormore Web and/or non Web application servers 102 (referred generally as“servers”), one or more client devices 106 and one or more networktraffic management devices 110, although the environment 100 can includeother numbers and types of devices in other arrangements. The networktraffic management device 110 is coupled to the servers 102 via localarea network (LAN) 104 and client devices 106 via a wide area network108. Generally, client device requests are sent over the network 108 tothe servers 102 which are received or intercepted by the network trafficmanagement device 110.

Client devices 106 comprise network computing devices capable ofconnecting to other network computing devices, such as network trafficmanagement device 110 and/or servers 102. Such connections are performedover wired and/or wireless networks, such as network 108, to send andreceive data, such as for Web-based requests, receiving server responsesto requests and/or performing other tasks. Non-limiting andnon-exhausting examples of such client devices 106 include personalcomputers (e.g., desktops, laptops), tablets, smart televisions, videogame devices, mobile and/or smart phones and the like. In an example,client devices 106 can run one or more Web browsers that provide aninterface for operators, such as human users, to interact with formaking requests for resources to different web server-based applicationsand/or Web pages via the network 108, although other server resourcesmay be requested by client devices.

The servers 102 comprise one or more server network devices or machinescapable of operating one or more Web-based and/or non Web-basedapplications that may be accessed by other network devices (e.g. clientdevices, network traffic management devices) in the environment 100. Theservers 102 can provide web objects and other data representingrequested resources, such as particular Web page(s), image(s) ofphysical objects, JavaScript and any other objects, that are responsiveto the client devices' requests. It should be noted that the servers 102may perform other tasks and provide other types of resources. It shouldbe noted that while only two servers 102 are shown in the environment100 depicted in FIG. 1B, other numbers and types of servers may beutilized in the environment 100. It is contemplated that one or more ofthe servers 102 may comprise a cluster of servers managed by one or morenetwork traffic management devices 110. In one or more aspects, theservers 102 may be configured implement to execute any version ofMicrosoft® IIS server, RADIUS server, DIAMETER server and/or Apache®server, although other types of servers may be used.

Network 108 comprises a publicly accessible network, such as theInternet, which is connected to the servers 102, client devices 106, andnetwork traffic management devices 110. However, it is contemplated thatthe network 108 may comprise other types of private and public networksthat include other devices. Communications, such as requests fromclients 106 and responses from servers 102, take place over the network108 according to standard network protocols, such as the HTTP, UDPand/or TCP/IP protocols, as well as other protocols. As per TCP/IPprotocols, requests from the requesting client devices 106 may be sentas one or more streams of data packets over network 108 to the networktraffic management device 110 and/or the servers 102. Such protocols canbe utilized by the client devices 106, network traffic management device110 and the servers 102 to establish connections, send and receive datafor existing connections, and the like.

Further, it should be appreciated that network 108 may include localarea networks (LANs), wide area networks (WANs), direct connections andany combination thereof, as well as other types and numbers of networktypes. On an interconnected set of LANs or other networks, includingthose based on differing architectures and protocols. Network devicessuch as client devices, 106, servers 102, network traffic managementdevices 110, routers, switches, hubs, gateways, bridges, cell towers andother intermediate network devices may act within and between LANs andother networks to enable messages and other data to be sent betweennetwork devices. Also, communication links within and between LANs andother networks typically include twisted wire pair (e.g., Ethernet),coaxial cable, analog telephone lines, full or fractional dedicateddigital lines including T1, T2, T3, and T4, Integrated Services DigitalNetworks (ISDNs), Digital Subscriber Lines (DSLs), wireless linksincluding satellite links and other communications links known to thoseskilled in the relevant arts. Thus, the network 108 is configured tohandle any communication method by which data may travel between networkdevices.

LAN 104 comprises a private local area network that allowscommunications between the one or more network traffic managementdevices 110 and one or more servers 102 in the secured network. It iscontemplated, however, that the LAN 104 may comprise other types ofprivate and public networks with other devices. Networks, includinglocal area networks, besides being understood by those skilled in therelevant arts, have already been generally described above in connectionwith network 108 and thus will not be described further.

As shown in the example environment 100 depicted in FIG. 1B, the one ormore network traffic management devices 110 is interposed between clientdevices 106 with which it communicates with via network 108 and servers102 with which it communicates with via LAN 104. In particular to thepresent disclosure, the network traffic management device 110 operatesin conjunction with a clustered multi-processing (CMP) system whichincludes one or more network traffic management devices 110, each ofwhich having one or more cores or processors 200 (FIG. 2A). Generally,the network traffic management device 110 manages networkcommunications, which may include one or more client requests and serverresponses, via the network 108 between the client devices 106 and one ormore of the servers 102. In any case, the network traffic managementdevice 110 may manage the network communications by performing severalnetwork traffic related functions involving the communications. Somefunctions include, but are not limited to, load balancing, accesscontrol, and validating HTTP requests using JavaScript code that aresent back to requesting client devices 106.

FIG. 2A is a block diagram of the network traffic management device inaccordance with an aspect of the present disclosure. As shown in FIG.2A, the example network traffic management device 110 includes one ormore device processors or cores 200, one or more device I/O interfaces202, one or more network interfaces 204, and one or more device memories206, which are coupled together by one or more bus 208. It should benoted that the network traffic management device 110 can be configuredto include other types and/or numbers of components and is thus notlimited to the configuration shown in FIG. 2A.

Device processor 200 of the network traffic management device 110comprises one or more microprocessors configured to executecomputer/machine readable and executable instructions stored in thedevice memory 206. Such instructions, when executed by one or moreprocessors 200, implement general and specific functions of the networktraffic management device 110, including the inventive process describedin more detail below. It is understood that the processor 200 maycomprise other types and/or combinations of processors, such as digitalsignal processors, micro-controllers, application specific integratedcircuits (“ASICs”), programmable logic devices (“PLDs”), fieldprogrammable logic devices (“FPLDs”), field programmable gate arrays(“FPGAs”), and the like. The processor 200 is programmed or configuredaccording to the teachings as described and illustrated herein.

Device I/O interfaces 202 comprise one or more user input and outputdevice interface mechanisms. The interface may include a computerkeyboard, mouse, display device, and the corresponding physical portsand underlying supporting hardware and software to enable the networktraffic management device 110 to communicate with other network devicesin the environment 100. Such communications may include accepting userdata input and providing user output, although other types and numbersof user input and output devices may be used. Additionally oralternatively, as will be described in connection with network interface204 below, the network traffic management device 110 may communicatewith the outside environment for certain types of operations (e.g. smartload balancing) via one or more network management ports.

Network interface 204 comprises one or more mechanisms that enable thenetwork traffic management device 110 to engage in networkcommunications over the LAN 104 and the network 108 using one or more ofa number of protocols, such as TCP/IP, HTTP, UDP, RADIUS and DNS.However, it is contemplated that the network interface 204 may beconstructed for use with other communication protocols and types ofnetworks. Network interface 204 is sometimes referred to as atransceiver, transceiving device, or network interface card (NIC), whichtransmits and receives network data packets over one or more networks,such as the LAN 104 and the network 108. In an example, where thenetwork traffic management device 110 includes more than one deviceprocessor 200 (or a processor 200 has more than one core), eachprocessor 200 (and/or core) may use the same single network interface204 or a plurality of network interfaces 204. Further, the networkinterface 204 may include one or more physical ports, such as Ethernetports, to couple the network traffic management device 110 with othernetwork devices, such as servers 102. Moreover, the interface 204 mayinclude certain physical ports dedicated to receiving and/ortransmitting certain types of network data, such as device managementrelated data for configuring the network traffic management device 110or client request/server response related data.

Bus 208 may comprise one or more internal device component communicationbuses, links, bridges and supporting components, such as bus controllersand/or arbiters. The bus 208 enables the various components of thenetwork traffic management device 110, such as the processor 200, deviceI/O interfaces 202, network interface 204, and device memory 206, tocommunicate with one another. However, it is contemplated that the bus208 may enable one or more components of the network traffic managementdevice 110 to communicate with one or more components in other networkdevices as well. Example buses include HyperTransport, PCI, PCI Express,InfiniBand, USB, Firewire, Serial ATA (SATA), SCSI, IDE and AGP buses.However, it is contemplated that other types and numbers of buses may beused, whereby the particular types and arrangement of buses will dependon the particular configuration of the network traffic management device110.

Device memory 206 comprises computer readable media, namely computerreadable or processor readable storage media, which are examples ofmachine-readable storage media. Computer readablestorage/machine-readable storage media may include volatile,nonvolatile, removable, and non-removable media implemented in anymethod or technology for storage of information. Examples of computerreadable storage media include RAM, BIOS, ROM, EEPROM, flash/firmwarememory or other memory technology, CD-ROM, digital versatile disks (DVD)or other optical storage, magnetic cassettes, magnetic tape, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store the information, which can be accessed by acomputing or specially programmed network device, such as the networktraffic management device 110.

Such storage media includes computer readable/processor-executableinstructions, data structures, program modules, or other data, which maybe obtained and/or executed by one or more processors, such as deviceprocessor 200. Such instructions, when executed, allow or cause theprocessor 200 to perform actions, including performing the inventiveprocesses described below. The memory 206 may contain other instructionsrelating to the implementation and operation of an operating system forcontrolling the general operation and other tasks performed by thenetwork traffic management device 110.

FIG. 2B illustrates a block diagram of the network interface inaccordance with an aspect of the present disclosure. In particular, FIG.2B shows the DMA processes used by network interface 204 for usingmultiple independent DMA channels with corresponding multipleapplications, where each application has its own driver, and for sendingpackets.

As illustrated in FIG. 2B, the host system 111 can send a network datapacket stored in host memory 212 to the network 108 via networkinterface controller 204 and Ethernet port 236. A send DMA operation isperformed when the host system 111 uses a DMA channel to move a block ofdata from host memory 22 to a network interface controller peripheral(not shown) via network 108. To perform a send DMA operation, the hostprocessor 200 places the target network data packet into DMA packetbuffer 216 and creates a DMA send descriptor (not shown separately) insend DMA descriptor rings 218. The DMA send descriptor is jointlymanaged by the host system 111 and the network interface controller 204.The DMA send descriptor includes an address field and length field. Theaddress field points to the start of the target network data packet inDMA packet buffer 216. The length field declares how many bytes oftarget data are present in the DMA packet buffer 216. The DMA senddescriptor also has a set of bit flags (not shown) used to signaladditional target data control and status information.

By way of example only, return DMA descriptor rings and send DMAdescriptor rings 218 can be physically in the same hardware memoryblocks functioning as return and send DMA rings, respectively, atdifferent times. Alternatively, separate and distinct memory blockswithin host memory 212 DMA memory resources 214 may be reserved for eachreturn DMA descriptor rings and send DMA descriptor rings 218.

Host system 111 places the send descriptor on the send DMA descriptorrings 218 in host system memory 212. The host processor 200 determinesthe QoS of the network packet to be transferred to the network 108 andmoves the network packet to the appropriate DMA packet buffer 216 andplaces the descriptor on the appropriate descriptor rings 1-4 in sendDMA descriptor rings 218. The descriptor ring in send DMA descriptorrings 218 is chosen by the host system 111 which selects the DMAchannel, its associated peripheral, and the QoS level within the DMAchannel. Send descriptors created by host system 111 in send DMAdescriptor rings 218 can be of variable types, where each descriptortype can have a different format and size. The send DMA descriptor rings218 are capable of holding descriptors of variable type.

The host processor 200 writes one or more mailbox registers 230 of thenetwork interface controller 204 to notify the network interfacecontroller 204 that the packet is ready. In performing thisnotification, the host processor 200 performs a write operation to amemory mapped network interface controller register (mailbox register230). The host processor 200 can report the addition of multipledescriptors onto the send DMA ring in a single update, or alternatively,in multiple updates.

The appropriate packet DMA engine within DMA engine 224 is notified thatthe packet is ready. The packet DMA engine 224 can be selected fromavailable DMA channels, or if a specific application has a dedicated DMAchannel, the associated packet DMA engine 224 for that channel is used.The DMA engine 224 retrieves the DMA descriptor from the send DMAdescriptor rings 218. When multiple descriptors are outstanding in thesend DMA descriptor rings 218, the DMA Engine 224 may retrieve more thanone descriptor. Retrieving multiple descriptors at a time maximizes busbandwidth and hardware efficiency. The DMA engine 224 is capable ofreceiving and processing send descriptors of variable type, format, andsize.

As outlined above, the packet DMA engine 224 monitors the progress ofthe host DMA operations via a set of mailbox registers 230. Each packetDMA engine 224 supports its own set of mailbox registers 230. Themailbox registers 230 reside in a mapped address space of the networkinterface controller 204. When appropriate, the host processor 200accesses the mailbox registers 230 by performing memory mapped read andwrite transactions to the appropriate target address. The mailboxregisters 230 also contain ring status information for the Ring to QoSMapper 228. In this send DMA example, the packet DMA engine 224 readsthe send descriptor, performs the DMA operation defined by it, andreports to the host system 111 that the DMA operation is complete.During the DMA operation, data is received from one or more CPU Bus readtransactions (e.g., HyperTransport or PCI Express read transactions).

DMA scheduler 226 chooses packets out of packet buffers 216 based uponthe priority of the queued network data packets and schedules thetransfer to the appropriate packet DMA engine 224. For clarity andbrevity, only a single packet buffer, a single DMA scheduler, and DMAengine are shown in FIG. 2B, but it should be understood that additionalpacket buffers, DMA schedulers, and DMA engines supporting theindependent DMA channels 1-n and associated applications App(1)-App(n)can be included in network interface controller 204.

The packet buffers 216 are selected based on the novel scheme (discussedbelow) using DMA scheduler 226. The DMA scheduler 226 selects whichdescriptor ring 1-4 out of return DMA descriptor rings (also referred toas return DMA rings, or send rings) within DMA memory resources 212 toservice and the matching packet buffer 216 is accessed for a singlepacket. The scheduling process is then repeated for the next packet.

Each network packet retrieved from a packet buffer 216 is routed to theappropriate DMA channel controlled by the respective packet DMA enginesuch as the packet DMA engine 224. The DMA channel segments the networkpacket for delivery to host memory 212 via several, smaller,HyperTransport packets. These HyperTransport packets are interleavedwith HyperTransport packets from the other DMA channels in the networkinterface controller 204.

Ring to QoS Mapper 228 examines the assigned send DMA ring in send DMAdescriptor rings 218 and receives packet data and packet controlinformation from the packet DMA engine 224. Using the controlinformation, the Ring to QoS Mapper 228 stamps the appropriate QoS ontothe network data packet, thereby allowing host system 111 to send thenetwork data packet back to the network 108. For example, using thecontrol information, the Ring to QoS Mapper 228 can create and prepend aHiGig header to the packet data.

An egress DMA routing interface 232 arbitrates access to the network forDMA send packets. When a Ring to QoS Mapper 228 has a network packetready to send, the egress DMA routing interface 232 arbitrates itsaccess to the Ethernet port 236 and routes the packet to the correctinterface if there is more than one present in the network interfacecontroller 204. The egress DMA routing interface 232 behaves like acrossbar switch and monitors its attached interfaces for availablepackets. When a packet becomes available, the egress DMA routinginterface 232 reads the packet from the selected ring to QoS mapper 228and writes it to the destination interface. The egress DMA routinginterface 232 moves complete packets to Ethernet MACs 234. When multiplesources are contending for egress DMA routing interface 232, the egressDMA routing interface 232 uses a weighted arbitration scheme asdiscussed in more detail below.

The network interface controller 204 provides DMA services to a hostcomplex such as the host system 111 on behalf of its attached I/Odevices such as the Ethernet port 236. DMA operations involve themovement of data between the host memory 212 and the network interfacecontroller 204. The network interface controller 204 creates and managesHyperTransport or other types of CPU Bus read/write transactionstargeting host memory 22. Data transfer sizes supported by DMA channelsmaintained by various components of application delivery controller 110are much larger than the maximum HyperTransport or CPU bus transactionsize. The network interface controller 204 segments single DMAoperations into multiple smaller CPU Bus or HyperTransport transactions.Additionally, the network interface controller 204 creates additionalCPU bus or HyperTransport transactions to support the transfer of datastructures between the network interface controller 204 and host memory212.

FIG. 2C illustrates further details of the network traffic managementdevice in accordance with an aspect of the present disclosure. Inparticular, the network traffic management device 110 is shown handlinga plurality of independent applications App(1)-App(n) being executed byone or more processors (e.g., host processor 200) in host system 111.Each application in the plurality of applications App(1)-App(n) has itsown respective application driver shown as Driver 1, Driver 2, . . . ,Driver ‘n’ associated with the respective application, where the index ndenotes an unlimited number of executing applications and drivers.Applications App(1)-App(n) send and receive data packets from and to thenetwork 108 (and/or LAN 104), respectively, using respective DMAchannels (e.g., DMA channels 1-n). DMA channels 1-n are uniquelyassigned to individual applications out of App(1)-App(n). In thisexample, drivers 1-n manage access to respective DMA channels 1-n and donot require knowledge of each other or a common management database orentity (e.g., a hypervisor). By way of example only, each ofapplications App(1)-App(n) can be independent instances of differentapplications, or alternatively, may be independent instances of the sameapplication, or further, may be different operating systems supported bydifferent processors in host system 111 (e.g., host processor 200).

DMA channels 1-n each have unique independent resources allotted tothem, for example, a unique PCI bus identity including a configurationspace and base address registers, an independent view of host systemmemory 212, a unique set of DMA descriptor ring buffers, a unique set ofpacket buffers 216, unique DMA request/completion signaling (throughinterrupts or polled memory structures), and other resources. Each ofDMA channels 1-n is unique and independent thereby permitting managementby separate unique drivers 1-n.

The network interface controller 204 classifies received packets todetermine destination application selected from applicationsApp(1)-App(n) and thereby selects the matching DMA channel to deliverthe packet to the corresponding application. By way of example only,packet classification includes reading packet header fields therebypermitting application identification. Further by way of example only,packet classification includes hash calculation for distribution ofpackets across multiple instances of the same application, and/orreading a cookie stored, for example, in the network interfacecontroller 204 associated with the application and the received networkpacket.

In general, the present disclosure utilizes hardware and software in thenetwork traffic management device when pacing network traffic being sentto another network device (e.g. client, server). The purpose is for thenetwork traffic management device to pace delivery of session data tothe client to match the rate at which the client consumes the data,which adds value for mobile clients and networks. The network trafficmanagement device utilizes a transmit time calendar having a pluralityof quanta of transmit times. For example, one quanta may be 1microsecond, although other time durations are contemplated.

FIG. 3 illustrates an example transmit ring or bucket in accordance withan aspect of the present disclosure. As shown in FIG. 3, the bucket 300is a software based data construct which holds a plurality of datapacket DMA descriptors which point to the pending transmit packets andhas packets from a plurality of various, different sessions. The bucket300 has a fence 302 which marks the last descriptor in the bucket.Software determines what data packets to send and when to send thosedata packets. The software communicates with the hardware componentwherein the software enhances the data structures to communicate a timecomponent with regard to when the selected data packets are to be sent.In particular, the software component divides session data into multiplemaximum segment sized (MSS) or smaller sized packets. The softwarecomponent distributes the packets in buckets separated by fixed transmittimes, wherein the buckets are then handed off to the hardwarecomponent, such as a DMA ring, to handle the delivery of the datapackets populated in the buckets, by processing the buckets themselves.

FIG. 4 illustrates a time line of an example transmit ring or bucket inaccordance with an aspect of the present disclosure. As can be seen inFIG. 4, the transmit time calendar utilizes a plurality of buckets 300where session data 400, such as data packet DMA descriptors is populatedinto the buckets 300. Each bucket has a predetermined size and adjacentbuckets are separate by a transmit time or quanta that is fixed (shownas T1-T4). It should be noted the time calendar is configured to have afinite number of buckets, wherein the calendar effectively wraps aroundto the first bucket after the last bucket has been handled. In addition,multiple packet descriptors can be placed in the same bucket to increasebandwidth. The software component may decide whether to pace a datapacket, as opposed to being bulk data (FIG. 5) based on size, type oftraffic the data packet is associated with, QoS parameters (based onflow) and the like.

In an aspect, one or more buckets 300 can be skipped for distribution,by the software component, to increase intra-packet spacing. One way fordetermining how many buckets to skip before placing a data packetdescriptor in a bucket 300 can be based on the type or class ofapplication in which the data is delivered (e.g. video streaming). Forexample, video streaming applications would benefit from a constant rateof data delivery. Another way of determining (based on speed) would beusing TCP congestion control algorithms which calculate how many packetsare to be sent over time. The software does not have to hunt for holeswhere packet descriptors can be inserted. Instead, the softwarecomponent need only to drop packet descriptors into the one or morebuckets 300.

Once a bucket 300 is populated by the software component, the softwarecomponent releases the bucket to the hardware component, such as thenetwork interface 204. In particular, the bucket contents are writteninto the network interface's 204 DMA ring 218 en-mass. In an aspect, thehardware component can determine whether the size of a particular datapacket in a bucket is of a threshold size such that the data packet canbe processed within the allotted time quanta.

A gross timer is applied by the hardware component in writing thepackets into the DMA ring 218, wherein the poll loop time of theprocessor 200 of the network traffic management device 110 is used asthe gross timer. The sum of released bucket time quanta's must excelpoll loop time.

The fence 304 marking at the end of a particular bucket 300 serves as atiming boundary at the end of the bucket to allow the hardware componentto do precise bucket to bucket timing.

FIG. 5 illustrates a block diagram of hardware based time enforcement inperformed by a high speed bridge (HSB) priority mechanism in accordancewith an aspect of the present disclosure. As shown in FIG. 5, datapackets that are written to buckets that are released to the hardwareare received in a pacing send ring 500. In contrast, non-paced data(data not written to buckets) are received in a bulk send ring 502.

In an aspect, the pacing send ring 500 is given higher priority in termsof being sent to the ring arbitrator 506. Thus, the bulk traffic in thebulk send ringer 502 advances to the arbitrator 506 when the pacedtraffic (from the pacing send ring 500) is blocked or is absent. Apacing timer 504 in coupled to the pacing send ring 500 and thearbitrator 506, wherein the pacing timer is reset at the start of a newbucket 300. The pacing timer 504 also blocks traffic at bucketboundaries until the bucket quanta time expires.

FIG. 6 illustrates an example transmit time line in accordance with anaspect of the present disclosure. As shown in the time line 600, thehardware component consumes the data packets in the first bucket 602 forthe entire time quanta which ends at boundary A. In the second bucket604, it is shown that the hardware component finishes writing the paceddata packets such that additional time remains for the allotted time.The hardware component thereafter processes bulk data for the remainingamount of time in the time quanta until reaching boundary B. The sameoccurs in bucket 606. In bucket 608, there is a period of time where nodata is written, which can result in no bulk data or paced data beingavailable for writing to the DMA engine.

FIG. 7 illustrates a flow chart of the software implementation ofpopulating buckets with data packet descriptors in accordance with anaspect of the present disclosure. As shown in FIG. 7, the process 700occurs when a first bucket of a plurality of buckets is selected by theapplication module 210 of the network traffic management device (Block702). The application module 210 thereafter populates the first bucketwith one or more data packet descriptors associated with data packets tobe released to the network interface 204 (Block 704). The applicationmodule 210 thereafter determines whether there are additional paced orbulk data packets that can be added to the current selected bucket(Block 706). If so, the process repeats back to Block 704. If not, theprocess proceeds to Block 708.

Once the application module 210 has populated the bucket with the lastdata packet, the application module 210 applies a fence which representsthe last data packet for that bucket (Block 708). The application module210 thereafter determines if the selected bucket is the last bucket inthe time calendar (Block 710). If so, the application module 210effectively wraps around the time calendar and selects the first bucket(Block 702). If not, the application module 210 selects the next bucketin the time calendar (Block 712), wherein the process proceeds back toBlock 704.

FIG. 8 illustrates a flow chart of the software implementation ofpopulating buckets with data packet descriptors in accordance with anaspect of the present disclosure. As shown in FIG. 8, the process 800occurs when a first bucket of a plurality of buckets is selected by thenetwork interface 204 of the network traffic management device (Block802). The network interface 204 thereafter processes paced datapacket(s) for the selected bucket (Block 804). The network interface 204thereafter determines whether additional time is available for theselected bucket (Block 806). If so, the network interface 204 determinesif bulk data packets are present in the bucket (Block 808). If so, thenetwork interface 204 processes the bulk data for the bucket (Block810). The process then repeats back to Block 806.

If no additional allotted time is left, the network interface 204selects the next bucket 812 and the process proceeds back to Block 804.Referring back to Block 808, if no bulk data is available for processingfor a particular bucket, the network interface 204 does not write anydata packets and allows the remaining time for the bucket quanta toexpire (Block 814). The process then proceeds to Block 812.

Having thus described the basic concepts, it will be rather apparent tothose skilled in the art that the foregoing detailed disclosure isintended to be presented by way of example only, and is not limiting.Various alterations, improvements, and modifications will occur and areintended to those skilled in the art, though not expressly statedherein. These alterations, improvements, and modifications are intendedto be suggested hereby, and are within the spirit and scope of theexamples. Additionally, the recited order of processing elements orsequences, or the use of numbers, letters, or other designationstherefore, is not intended to limit the claimed system and/or processesto any order except as may be specified in the claims. Accordingly, thesystem and method is limited only by the following claims andequivalents thereto.

What is claimed is:
 1. A method for transmitting data packets at anoptimized rate, the method comprising: populating, by the networktraffic management computing device, a plurality of buckets with one ormore selected data packet descriptors associated with one or morecorresponding ones of a subset of a plurality of data packets to betransmitted as paced and another subset of the data packets to betransmitted as bulk; releasing, by the network traffic managementcomputing device, one of the buckets to a hardware component comprisinga pacing send ring and a bulk send ring, the releasing comprisingwriting one or more of the subset of the data packets to the pacing sendring and one or more of the another subset of the data packets to thebulk send ring; and transmitting, by the network traffic managementcomputing device, the one or more of the subset of the data packets fromthe pacing send ring and the one or more of the another subset of thedata packets from the bulk send ring for a predetermined transmit time,wherein the subset of the data packets are strictly prioritized over theanother subset of the data packets and the one or more of the anothersubset of the data packets are only transmitted within the predeterminedtransmit time when the pacing send ring is emptied during thepredetermined transmit time.
 2. The method as set forth in claim 1,further comprising repeating, by the network traffic managementcomputing device, the releasing and transmitting for each other of thebuckets.
 3. The method as set forth in claim 1, wherein the one or moreof the subset of the data packets from the pacing send ring and the oneor more of the another subset of the data packets from the bulk sendring are transmitted by a direct memory access (DMA) transmit engine. 4.The method as set forth in claim 1, wherein the subset of the datapackets to be transmitted is identified as paced and the another subsetof the data packets is identified as bulk based on a data packet size,an associated type of traffic, or a quality of service (QoS) parameter.5. The method as set forth in claim 1, further comprising: determining,by the network traffic management computing device, when another one ofthe buckets should be skipped based on a type of application from whichthe ones of the data packets associated with the selected data packetdescriptors populated therein originated; and skipping, by the networktraffic management computing device, the another one of the buckets suchthat the releasing and transmitting are not repeated for the another oneof the buckets, when the determining indicates that the another one ofthe buckets should be skipped.
 6. The method as set forth in claim 1,further comprising waiting, by the network traffic management computingdevice, to release another one of the buckets when the one or more ofthe subset of the data packets are transmitted from the pacing send ringand the one or more of the another subset of the data packets aretransmitted from the bulk send ring prior to the expiration of thepredetermined transmit time.
 7. A non-transitory computer readablemedium having stored thereon instructions for transmitting data packetsat an optimized rate, comprising executable code which when executed byat least one processor and/or network interface causes the processorand/or network interface to perform steps comprising: populating aplurality of buckets with one or more selected data packet descriptorsassociated with one or more corresponding ones of a subset of aplurality of data packets to be transmitted as paced and another subsetof the data packets to be transmitted as bulk; releasing one of thebuckets to a hardware component comprising a pacing send ring and a bulksend ring, the releasing comprising writing one or more of the subset ofthe data packets to the pacing send ring and one or more of the anothersubset of the data packets to the bulk send ring; and transmitting theone or more of the subset of the data packets from the pacing send ringand the one or more of the another subset of the data packets from thebulk send ring for a predetermined transmit time, wherein the subset ofthe data packets are strictly prioritized over the another subset of thedata packets and the one or more of the another subset of the datapackets are only transmitted within the predetermined transmit time whenthe pacing send ring is emptied during the predetermined transmit time.8. The non-transitory computer readable medium as set forth in claim 7,wherein the executable code when executed by the processor and/or thenetwork interface further causes the processor and/or the networkinterface to perform at least one additional step comprising repeatingthe releasing and transmitting for each other of the buckets.
 9. Thenon-transitory computer readable medium as set forth in claim 7, whereinthe one or more of the subset of the data packets from the pacing sendring and the one or more of the another subset of the data packets fromthe bulk send ring are transmitted by a direct memory access (DMA)transmit engine.
 10. The non-transitory computer readable medium as setforth in claim 7, wherein the subset of the data packets to betransmitted is identified as paced and the another subset of the datapackets is identified as bulk based on a data packet size, an associatedtype of traffic, or a quality of service (QoS) parameter.
 11. Thenon-transitory computer readable medium as set forth in claim 7, whereinthe executable code when executed by the processor and/or the networkinterface further causes the processor and/or the network interface toperform at least one additional step comprising: determining whenanother one of the buckets should be skipped based on a type ofapplication from which the ones of the data packets associated with theselected data packet descriptors populated therein originated; andskipping the another one of the buckets such that the releasing andtransmitting are not repeated for the another one of the buckets, whenthe determining indicates that the another one of the buckets should beskipped.
 12. The non-transitory computer readable medium as set forth inclaim 7, wherein the executable code when executed by the processorand/or the network interface further causes the processor and/or thenetwork interface to perform at least one additional step comprisingwaiting to release another one of the buckets when the one or more ofthe subset of the data packets are transmitted from the pacing send ringand the one or more of the another subset of the data packets aretransmitted from the bulk send ring prior to the expiration of thepredetermined transmit time.
 13. A network traffic management computingdevice comprising at least one processor and/or network interface and amemory coupled to the processor and/or network interface which isconfigured to be capable of executing programmed instructions comprisingand stored in the memory to: populate a plurality of buckets with one ormore selected data packet descriptors associated with one or morecorresponding ones of a subset of a plurality of data packets to betransmitted as paced and another subset of the data packets to betransmitted as bulk; release one of the buckets to a hardware componentcomprising a pacing send ring and a bulk send ring, the releasingcomprising writing one or more of the subset of the data packets to thepacing send ring and one or more of the another subset of the datapackets to the bulk send ring; and transmit the one or more of thesubset of the data packets from the pacing send ring and the one or moreof the another subset of the data packets from the bulk send ring for apredetermined transmit time, wherein the subset of the data packets arestrictly prioritized over the another subset of the data packets and theone or more of the another subset of the data packets are onlytransmitted within the predetermined transmit time when the pacing sendring is emptied during the predetermined transmit time.
 14. The networktraffic management computing device as set forth in claim 13, whereinthe processor and/or network interface coupled to the memory is furtherconfigured to be capable of executing at least one additional programmedinstruction comprising and stored in the memory to repeat the releasingand transmitting for each other of the buckets.
 15. The network trafficmanagement computing device as set forth in claim 13, wherein the one ormore of the subset of the data packets from the pacing send ring and theone or more of the another subset of the data packets from the bulk sendring are transmitted by a direct memory access (DMA) transmit engine.16. The network traffic management computing device as set forth inclaim 13, wherein the subset of the data packets to be transmitted isidentified as paced and the another subset of the data packets isidentified as bulk based on a data packet size, an associated type oftraffic, or a quality of service (QoS) parameter.
 17. The networktraffic management computing device as set forth in claim 13, whereinthe processor and/or network interface coupled to the memory is furtherconfigured to be capable of executing at least one additional programmedinstruction comprising and stored in the memory to: determine whenanother one of the buckets should be skipped based on a type ofapplication from which the ones of the data packets associated with theselected data packet descriptors populated therein originated; and skipthe another one of the buckets such that the releasing and transmittingare not repeated for the another one of the buckets, when thedetermining indicates that the another one of the buckets should beskipped.
 18. The network traffic management computing device as setforth in claim 13, wherein the processor and/or network interfacecoupled to the memory is further configured to be capable of executingat least one additional programmed instruction comprising and stored inthe memory to wait to release another one of the buckets when the one ormore of the subset of the data packets are transmitted from the pacingsend ring and the one or more of the another subset of the data packetsare transmitted from the bulk send ring prior to the expiration of thepredetermined transmit time.